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ABSTRACT 


The  Air  Force  has  expressed  interest  in  a 
tactically  effective  hyperspectral  imaging 
payload  for  use  in  a  rapid  response 
demonstration  program.  The  goals  set  for  a 
high  quality  imaging  spectrometer  payload 
includes  radiometric  performance  (sensitivity, 
saturation  range,  quantization  level,  and 
absolute  radiometry  accuracy),  spectral 
performance  (spectral  range,  resolution, 
sampling,  and  spectral  calibration  accuracy), 
spatial  performance  (image  swath  width, 
image  length,  spatial  resolution,  ground 
sample  distance,  and  geolocation  accuracy)  as 
well  as  mission  constraints  related  to  payload 
weight,  volume,  power,  and  data  rate. 
Raytheon  has  developed  a  number  of  design 
tools  needed  to  balance  these  requirements 
using  real-world  hyperspectral  detection  case 
studies  that  relate  directly  to  military  tactical 
effectiveness.  These  techniques  make  use  of 
high  quality  imaging  spectrometer  data  sets 
from  airborne  collects  to  enable  evaluation  of 
probability  of  detection  versus  probability  of 
false  alarm  for  arbitrary  spectral  target  sets.  In 
this  way,  reasonable  surrogate  metrics  for 
tactical  military  utility  may  be  examined  as  a 
function  of  payload  performance  requirements 
in  order  to  perform  meaningful  cost-as-an- 
independent-variable  (CAIV)  analysis. 


Copyright  notice  .... 


INTRODUCTION 

Recent  Responsive  Space  Missions  call  for 
achieving  80%  of  mission  performance  for 
20%  of  the  cost.  Applying  this  challenge  to 
development  of  a  hyperspectral  imaging 
payload  for  a  responsive  space  mission 
demands  a  more  quantitative  means  to  relate 
cost-performance  trades  to  measures  of 
effectiveness  directly  related  to  military 
utility.  This  paper  presents  a  simulation 
strategy  that  leverages  existing  high  quality 
airborne  data  sets  and  hyperspectral  design 
and  simulation  tools  to  meet  this  challenge. 


Figure  1.  Notional  plot  of  performance  versus  cost 
(Adapted  from  Wertz,  1999) 

The  notional  plot  of  performance  versus  cost, 
shown  in  Figure  L,  is  a  simple  and  effective 
way  to  illustrate  the  idea  of  an  optimum  point 
along  the  curve1.  Best  performance  solutions 


1  Wertz,  James  R.,  Wiley  J.  Larson,  eds.  1999,  Space 
Mission  Analysis  and  Design,  El  Segundo  CA, 
Microcosm  Press 
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are  typically  not  affordable  for  Responsive 
Space  missions,  while  lowest  cost  missions 
have  little  military  utility.  Some  where 
between  the  two  there  should  exist  an 
optimum  point,  either  in  the  sense  of  best 
performance  per  unit  cost,  or  in  a  Pareto  sense 
with  a  utility  near  80%  of  the  best 
performance  solution  at  20%  of  the  cost.  To 
find  the  optimum  cost-performance  point  it  is 
necessary  to  understand  how  dimensions  of 
payload  performance  relate  to  quantitative 
measures  of  military  utility,  and  how  costs  are 
estimated  for  those  levels  of  aggregate 
performance. 

Feasible  combinations  of  payload  cost  and 
performance  can  be  obtained  from  competing 
payload  suppliers  or  by  one  of  several  cost 
estimating  techniques.  Analogy-based 
methods  compare  the  new  mission  to 
historical  cost  data  for  recent  similar  missions. 
Adjustments  in  cost  are  made  for  the  relative 
complexity  or  performance  required  by  the 
new  payload  as  compared  to  the  historical 
examples.  Parametric  methods  also  make  use 
of  historical  data  in  the  form  of  a  multiple 
regression  against  cost-driving  payload 
parameters.  One  challenge  with  using  these 
methods  to  estimate  cost  for  a  Responsive 
Space  Payload  is  the  lack  of  suitable  historical 
data.  Comparisons  to  historical  data  are 
limited  to  just  a  few  attributes  such  as  weight, 
aperture  size,  and  spectral  range  (visible  or 
infrared).  It  is  difficult  to  explore  meaningful 
trades  in  payload  performance  attributes  using 
analogy-based  and  parametric-based  methods. 
A  bottoms-up  detailed  cost  estimate,  likely  to 
be  the  method  used  by  payload  suppliers 
responding  to  an  RFP,  provides  good  cost 
insight  for  a  point  design  solution.  The  point 
design  may  or  may  not  be  the  desired  Pareto 
solution. 

Comparing  between  different  point  designs 
with  different  sets  of  performance  attributes 
requires  a  means  to  assess  aggregate 


performance  as  it  relates  to  military  utility. 
The  dimensions  of  performance  for  a 
hyperspectral  payload  apply  to  the  spectral, 
spatial,  and  radiometric  dimensions  of  the 
output  data  product.  For  each  dimension  of  the 
data  cube,  the  performance  specifies  the 
range,  resolution,  sampling,  and  knowledge 
accuracy  of  the  resultant  data.  Any  single 
element  in  the  output  data  cube  exhibits 
spectral  and  spatial  resolution  attributes  and 
the  distance  between  adjacent  imaging  cube 
data  elements  defines  the  spectral  and  spatial 
sampling.  The  radiometric  sampling  is  defined 
by  the  data  quantization  level,  while  the  Noise 
Equivalent  Spectral  Radiance  (NESR)  defines 
the  radiometric  resolution.  The  outside 
dimensions  of  the  data  cube  define  the  spectral 
range  and  spatial  range  relates  to  image  swath 
width  and  down-track  image  length. 
Radiometric  range  is  set  by  the  maximum 
radiance  at  which  the  sensor  is  still  below 
saturation.  Calibration  performance 

parameters  define  the  accuracy  with  which 
absolution  values  may  be  placed  on  any  of  the 
performance  dimensions  and  specifically 
includes  spectral  calibration,  geolocation 
accuracy,  and  radiometric  accuracy.  Other 
performance  parameters  limit  non-linearity  in 
response  to  single  performance  dimensions 
and  non-orthogonal  interactions  between 
performance  dimensions,  and  include 
temporal  slew  effects,  stray-light,  electronic 
cross-talk,  and  control  of  spectral/spatial  non¬ 
uniformities.  Finally,  other  performance 
dimensions  relate  to  payload  constraints  such 
as  weight,  volume,  power,  data  rate,  and  data 
volume. 

The  traditional  trade  study  is  one  way  to 
evaluate  overall  utility  between  point  designs 
with  differing  sets  of  performance  attributes. 
Ideally,  a  trade  study  compares  just  a  few 
candidate  options.  Go/no  go  criteria  are 
established  along  performance  dimensions, 
and  a  qualitative  weighting  is  developed  for 
each  dimension,  and  another  weighting  is 
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developed  between  dimensions.  A  second 
method  evaluating  overall  utility  is  to  form  the 
product  of  scaled  performance  dimensions.  An 
example  of  a  scaled  dimension  is  the  ratio 
between  the  achieved  Swath  width  and  a  goal 
or  threshold  swath  width.  In  a  case  such 
NESR,  where  higher  performance  comes  from 
a  smaller  value,  the  ratio  is  inverted.  Each 
performance  dimension  requires  a  goal  or 
threshold  value,  and  the  overall  performance 
metric  is  the  product  of  the  scaled 
performance  dimensions. 

While  both  these  techniques  are  effective  for 
selecting  a  design  option,  they  depend  on 
qualitative  weighting  factors  and  goal 
performance  values  with  only  indirect  linkage 
to  mission  military  utility.  Many  performance 
dimensions  are  highly  non-linear  in  utility.  For 
example,  Swayze  shows  that  finer  spectral 
resolution  improves  overall  utility  up  to  the 
point,  and  then  flattens2.  Passing  this  point  of 
optimum  spectral  resolution  may  then  have  an 
adverse  impact  on  NESR.  It  is  difficult  to 
capture  all  these  subtle  effects  using  trade 
studies  and  product  of  scaled  performance 
metrics.  This  insight  has  led  Raytheon  to 
develop  a  more  direct  means  by  which  to 
assess  complex  interactions  between 
performance  dimensions  in  a  way  that  can  be 
directly  linked  to  military  utility. 

UTILITY  ANALYSIS  SIMULATION 

The  spectral  radiance  that  enters  a 
hyperspectral  sensor  can  be  decomposed  into 
elements  of  target  and  background  spectral 
reflectance,  atmospheric  radiative  transfer 
processes  and  solar  illumination  and  viewing 
geometry.  The  response  to  that  input  radiance 
results  in  a  raw  output  signal.  The  sensor 

2  Swayze.G.  A.,  R.  N.  Clark,  A.  F.  H.  Goetz,  T.  G. 
Chrien,  and  N.  S.  Gorelick,  Effects  of  spectrometer 
band  pass,  sampling,  and  signal-to-noise  ratio  on 
spectral  identification  using  the  Tetracorder  algorithm, 

J.  Geoph.  Research  (Planets),  vol.  108,  No.  E9, 5105, 
doi:  10.1029/2002JE001975, 2003. 


response  function  may  be  decomposed  into 
the  performance  dimensions  that  are  the  topic 
of  this  discussion.  The  output  signal  is 
calibrated,  atmospherically  corrected,  and  then 
processed  by  algorithms  to  produce 
information  potentially  useful  to  a  military 
need.  These  process  steps  are  depicted  in 
Figure  2.  By  simulating  this  chain  of  events  it 
is  possible  to  form  the  relationship  between 
the  performance  dimensions  that  describe  the 
sensor  response  function  and  end  measures  of 
military  utility. 
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Figure  2.  Process  steps  used  to  relate  sensor 
performance  to  military  utility 


Most  of  the  process  steps  described  above  are 
simulated  with  available  modeling  tools.  The 
Modtran  code  is  employed  to  simulate  at- 
sensor  input  radiance  from  input  reflectance, 
viewing  and  illumination  geometries,  and 
atmospheric  parameters.  Detail  sensor 
response  models  have  been  written  which 
simulate  the  manner  by  which  the  sensor 
degrades  the  input  radiance  spectrally  and 
spatially,  and  adds  noise  and  the  effects  of 
pixel-by-pixel  variations  in  gain  and  offset. 
Calibration,  atmospheric  correction  and 
information  algorithms  exist  and  are  well 
understood.  Metrics  of  military  utility,  such  as 
the  probability  of  false  alarm  given  a  required 
probability  of  detection  are  also  well 
established. 


The  new  innovation  is  to  form  a  sequence  of 
these  models  in  order  to  investigate  the  direct 
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military  utility  of  different  combinations  of 
performance  dimensions.  High  quality 
airborne  hyperspectral  image  data  is  available 
to  serve  as  realistic  scene  backgrounds.  By 
inverting  this  data  to  surface  reflectance  using 
commercially-available  atmospheric  inversion 
codes,  a  large  set  of  real  background  spectra 
are  made  available.  Targets  of  military  interest 
may  be  inserted  into  these  backgrounds  with  a 
perfect  knowledge  of  ground  truth  location, 
and  may  be  mixed  with  background  spectra  at 
fractions  between  zero  and  one.  MODTRAN 
is  used  to  simulate  at-sensor  radiance  at  very 
high  spectral  resolution  and  sampling. 

The  next  step  is  to  process  the  simulated  input 
radiance  with  sensor  response  models  based 
on  the  attributes  of  performance  under  study. 
Ideally  the  airborne  data  is  collected  at  higher 
spatial  resolution  than  is  envisioned  for  the 
mission  under  study  so  the  full  spatial  and 
spectral  effects  from  the  sensor  response 
function  are  captured.  An  optical  ray  trace 
model  generates  the  point  spread  functions 
including  the  impact  of  aberrations  and 
diffraction  from  the  optical  design  and  wave 
front  error  to  account  for  manufacturing  and 
alignment  tolerance.  The  effects  of 
spectrometer  entrance  slit  and  detector  width 
are  combined  with  detector  diffusion  effects 
and  scattering  models  in  order  to  estimate 
system  spectral  and  spatial  response  blurring 
kernels.  The  estimates  are  made  for  a 
sampling  of  points  over  the  spatial  and 
spectral  range  of  response  of  the  sensor,  and 
interpolations  of  these  responses  are  used  for 
points  in  between. 

First  order  area-solid  angle  product  and 
transmittances  are  included  to  capture  the 
details  of  the  radiometric  throughput.  Detector 
noise,  gain,  and  offset  variations  are  modeled 
or  input  from  detector  measurement,  if 
available.  High  resolution  input  radiance  is 
convolved  with  the  sensor  response  functions. 


and  a  simulated  output  signal  data  set  with 
noise  and  calibration  variations  is  generated. 

The  simulated  output  signal  from  the  sensor 
response  model  is  then  calibrated  back  to  units 
of  spectral  radiance  using  standard  calibration 
equations.  Since  the  calibration  parameters  are 
used  in  the  conversion  from  input  radiance  to 
output  signal,  it  is  easy  to  simulate  the  effect 
of  small  errors  in  the  calibration  process  used 
to  reverse  the  effect.  The  simulated  output 
radiance  thereby  contains  all  the  effects  of  the 
sensor  performance  attributes  including  the 
spectral  spatial,  and  radiometric  range, 
resolution,  sampling,  and  calibration 
inaccuracies. 

The  next  step  in  the  process  is  to  implement  a 
standard  atmospheric  correction.  This  results 
in  a  data  set  of  the  targets  and  backgrounds  in 
reflectance  space,  similar  to  input  reflectance 
used  in  the  first  step,  but  degraded  by  sensor 
resolution  and  containing  all  the  noise  and 
calibration  errors  of  the  sensor  response 
model.  This  data  is  input  to  standard  target 
detection  algorithms.  Performance  metrics 
such  as  the  probabilities  of  detecting  a  target 
or  of  false  alarm  detection  are  easy  to  compute 
since  the  ground  truth  is  perfectly  controlled 
in  the  simulation.  These  detection  metrics  may 
be  directly  related  to  issues  of  military  utility. 

Baseline  results  for  the  simulation  are  made 
by  comparing  the  metric  result  for  simulated 
targets  in  background  prior  to  and  after 
application  of  the  sensor  response  model.  This 
determines  the  impact  to  the  military  utility 
metric  by  the  set  of  performance  parameters 
captured  in  the  sensor  response  model.  The 
baseline  represents  the  best  the  sensor  can 
achieve  and  occupies  the  right-most  point  on 
the  performance-versus-cost  curve. 
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APPLICATION  OF  THE  SIMULATION 


SUMMARY 


The  end  purpose  of  the  simulation  is  to  fill  in 
points  along  the  performance-versus-cost 
curve  so  that  an  optimum  solution  may  be 
identified  and  selected.  In  order  to  properly  tie 
results  to  pertinent  military  utility  it  is 
important  to  understand  and  document  the 
customer’s  unique  definition  of  utility  and  to 
select  a  suitable  set  of  targets,  background, 
atmospheres,  and  illumination/viewing 
geometries.  Also,  it  is  important  to  adopt 
detection  (information  extraction)  algorithms 
that  are  the  same  of  similar  to  what  the 
customer  expects  to  employ.  Once  this  is 
completed,  it  is  important  to  examine  the 
baseline  case  (ideal  sensor  response,  no  noise, 
no  calibration  errors)  to  ensure  that  this  case 
provides  the  desired  utility  without  regard  to 
cost.  Then  it  is  possible  to  examine  real  point 
design  options  (sets  of  performance  attributes) 
using  the  utility  simulation  and  cost  estimating 
techniques  to  place  these  point  of  the 
performance-versus-cost  curve. 

Raytheon  is  working  with  a  local  college  to 
develop  the  military  utility  metric  analysis 
tool  set  using  a  very  simple  spectral  angle 
mapper  algorithm3 * 5  and  surrogate  targets  and 
backgrounds.  The  work  is  being  conducted  as 
a  senior  year  Engineering  Clinic  by  a  team  of 
five  students.  Initial  results  of  this  effort  have 
been  used  to  examine  the  utility  based 
performance-versus-cost  for  spectral  band 
range  trades,  and  to  examine  the  benefits  of 
atmospheric  correction. 


3  Kruse,  F.A.,  Boardman,  J.W.,  Lefkoff,  A.B., 

Heidebrecht,  K.B.,  Shapiro,  A.T.,  Barloon,  P.J.,  and 
Goetz,  A.F.H.  (1993).  The  Spectral  Image  Processing 
System  (SIPS)  Interactive  Visualization  and  Analysis  of 
Imaging  Spectrometer  Data.  Remote  Sensing  of 
Environment,  Vol.  44,  p.145-163. 


This  paper  presents  a  simulation  strategy  for 
relating  cost/performance  trades  to  military 
utility.  The  aggregate  impact  of  multiple 
dimensions  of  sensor  performance  in  a  given 
point-design  may  be  assessed  and  compared 
against  any  other  point  design.  The 
performance  impact  is  easy  to  baseline  against 
an  idealized  “highest  cost”  sensor  solution  by 
using  an  ideal  or  error-free  sensor  response 
function.  Performance  impact  may  be 
parameterized  along  any  given  performance 
dimension  and  related  to  a  military  utility 
metric.  It  is  important  to  involve  all  stake 
holders  in  the  defining  the  military  utility 
metrics  to  be  used,  and  to  agree  on  the  range 
of  targets,  backgrounds,  atmospheres, 
illumination/viewing  geometries,  and 
algorithms  to  be  used. 
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